Skip to content

Update Kimi-K2.6 B300 NVFP4 recipe / 更新 Kimi-K2.6 B300 NVFP4 配置#626

Open
Ankur-singh wants to merge 1 commit into
vllm-project:mainfrom
Ankur-singh:recipe/kimi-k2.6-nvfp4
Open

Update Kimi-K2.6 B300 NVFP4 recipe / 更新 Kimi-K2.6 B300 NVFP4 配置#626
Ankur-singh wants to merge 1 commit into
vllm-project:mainfrom
Ankur-singh:recipe/kimi-k2.6-nvfp4

Conversation

@Ankur-singh

@Ankur-singh Ankur-singh commented Jul 11, 2026 •
edited
Loading

Copy link
Copy Markdown

Summary

Validation

  • node scripts/build-recipes-api.mjs
    • ✓ JSON API: 147 models ... 8 strategies, 2 kv-store deployments, 2 platforms
  • generated-command and API checks confirm:
    • B300 NVFP4 uses TP4, Eagle3 draft length 4, and TOKENSPEED_MLA
    • the shared Simple option emits exactly one SimpleCPUOffloadConnector
    • production commands omit rejection_sample_method and synthetic_acceptance_length
    • DCP remains guide-only
    • public/nvidia/Kimi-K2.6-NVFP4.json retains the pinned nightly and TP4 metadata
  • source InferenceX sweep: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29158176591

Source

中文说明

验证

  • node scripts/build-recipes-api.mjs
    • ✓ JSON API: 147 models ... 8 strategies, 2 kv-store deployments, 2 platforms
  • 生成命令与 API 检查确认：
    • B300 NVFP4 使用 TP4、Eagle3 draft length 4 和 TOKENSPEED_MLA
    • 统一的 Simple 选项只生成一个 SimpleCPUOffloadConnector
    • 生产命令不包含 rejection_sample_method 和 synthetic_acceptance_length
    • DCP 继续仅在指南中说明，不通过命令生成器暴露
    • public/nvidia/Kimi-K2.6-NVFP4.json 保留固定的 nightly 镜像和 TP4 元数据
  • InferenceX 源扫描运行：https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29158176591

cc @faradawn for review

@vercel

vercel Bot commented Jul 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
vllm-recipes Ready Ready Preview, Comment Jul 13, 2026 9:00pm

Request Review

gemini-code-assist[bot]
gemini-code-assist Bot reviewed Jul 11, 2026
View reviewed changes

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the Kimi-K2.6 model configuration to support NVIDIA B300 and Blackwell GPUs, bumping the minimum vLLM version to 0.25.0, pinning the NVIDIA docker image to a nightly build, and introducing a native CPU KV offload feature. The nvfp4 variant is updated with optimized parameters, and the markdown guide is expanded to document these new serving paths. Feedback suggests adding VLLM_USE_SIMPLE_KV_OFFLOAD: "1" to the nvfp4 variant's extra_env so that the environment variable is correctly generated in the UI command when the CPU KV offload feature is enabled.

Comment thread models/moonshotai/Kimi-K2.6.yaml
@Ankur-singh Ankur-singh mentioned this pull request Jul 11, 2026
Open
ivanium
ivanium reviewed Jul 11, 2026
View reviewed changes

@ivanium ivanium left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good and self contained. We can remove the synthetic spec decoding args.

Comment thread models/moonshotai/Kimi-K2.6.yaml Outdated
Comment thread models/moonshotai/Kimi-K2.6.yaml Outdated
Comment thread models/moonshotai/Kimi-K2.6.yaml Outdated
@Ankur-singh
docs: update Kimi-K2.6 B300 recipe
4c79d09 
Document the validated NVFP4 EAGLE3, DCP, and native CPU KV offload paths from InferenceX PR #2158.

Use the repository-wide Simple KV Offload row and keep benchmark-only synthetic acceptance controls out of the serving recipe.

中文：补充 InferenceX PR #2158 已验证的 Kimi-K2.6 NVFP4 B300 EAGLE3、解码上下文并行和原生 CPU KV 缓存卸载配置；复用仓库统一的 Simple KV Offload 入口，并从生产推理配置中移除仅用于基准测试的合成接受率参数。

Signed-off-by: Ankur-singh <ankusingh@nvidia.com>
@Ankur-singh Ankur-singh force-pushed the recipe/kimi-k2.6-nvfp4 branch from 9446432 to 4c79d09 Compare July 13, 2026 20:58
@Ankur-singh ChatGPT Codex Connector

Ankur-singh commented Jul 13, 2026

Copy link
Copy Markdown
Author

@ivanium @faradawn The review feedback has been addressed and all four threads are resolved. Fresh validation with node scripts/build-recipes-api.mjs reports 147 models, 8 strategies, 2 KV-store deployments, and 2 platforms. The shared KV Offload row emits SimpleCPUOffloadConnector, and the production Eagle3 commands no longer contain the synthetic acceptance controls. Could you please re-review?

中文：@ivanium @faradawn 评审反馈已全部处理，四个评审线程也已解决。重新运行 node scripts/build-recipes-api.mjs 后，验证结果为 147 个模型、8 种策略、2 个 KV 存储部署和 2 个平台。统一的 KV Offload 配置会生成 SimpleCPUOffloadConnector，生产环境的 Eagle3 命令也已移除合成接受率控制参数。请帮忙重新评审。

esmeetu
esmeetu reviewed Jul 14, 2026
View reviewed changes
Comment thread models/moonshotai/Kimi-K2.6.yaml
parameter_count: "1T"
active_parameters: "32B"
context_length: 262144
supports_dcp: true

@esmeetu esmeetu Jul 14, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we keep this config for Advance feature:

Image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@esmeetu esmeetu esmeetu left review comments
@ivanium ivanium ivanium left review comments
+1 more reviewer
@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Ankur-singh @esmeetu @ivanium